SPECIFICATION 
TITLE 

"IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD 
AND STORAGE MEDIUM" 

BACKGROUND OF THE INVENTION 

The present invention relates to an image processing apparatus, an image 
processing method and a storage medium. More particularly, the invention relates to 
an image processing apparatus, an image processing method and a storage medium for 
creating a sharply- focused image by the use of a plurality of images having different 
degrees of defocusing. 

Today, so-called auto-focus cameras are becoming the mainstream of 
photographic apparatuses. In operation, the auto-focus camera measures the distance 
to a target object, moves the lens into an optimum position relative to the measured 
distance (i.e., a lens position such that a focused image can be projected onto an image 
pickup plane), and takes a focused picture of the object. 

A major disadvantage of the auto-focus cameras is their high price. Expensive 
parts are necessary for moving the lens to achieve focus. 

The present invention has been made in view of the above circumstances and 
provides an apparatus and a method constituting a low-cost scheme that creates 
focused images of target objects without moving the lens. 

SUMMARY OF THE INVENTION 
In carrying out the invention and according to one aspect thereof, there is 
provided an image processing apparatus for processing images picked up on a number 
of different image pickup planes, each having a different distance to a lens. The 
apparatus includes: an acquiring element for acquiring a plurality of images of a 



target object picked up on the plurality of image pickup planes; a first computing 
element for computing weights of the plurality of images acquired by the acquiring 
element; a second computing element for computing averages of the plurality of 
images acquired by the acquiring element; and an image creating element for creating 

5 an optimum image of the target object by obtaining a weighted average based on the 
weights computed by the first computing element and on the averages of the plurality 
of images computed by the second computing element. 

In a preferred embodiment, the image processing apparatus further includes an 
image pickup element for picking up images of the target object on the multiple 

1 0 different image pickup planes. 

According to another aspect of the invention, there is provided an image 
processing method for use with an image processing apparatus for processing images 
picked up on a number of different image pickup planes each having a different 
distance to a lens. The method includes the steps of acquiring multiple images of a 

1 5 target object picked up on the multiple different image pickup planes; computing 

weights of the different images acquired in the acquiring step; computing averages of 
the multiple images acquired in the acquiring step; and creating an optimum image of 
the target object by obtaining a weighted average based on the weights computed in 
the first computing step and on the averages of the multiple images computed in the 

20 second computing step. 

According to a further aspect of the invention, there is provided a storage 
medium which stores a computer-readable program for use with an image processing 
apparatus for processing images picked up on a number of different image pickup 



planes having different distances to a lens. The program embodies the steps of 
acquiring a number of different images of a target object picked up on the multiple 
different image pickup planes; computing weights of the multiple images acquired in 
the acquiring step; computing averages of the multiple images acquired in the 
5 acquiring step; and creating an optimum image of the target object by obtaining a 
weighted average based on the weights computed in the first computing step and on 
the averages of the multiple images computed in the second computing step. 

Through the use of the inventive image processing apparatus, image 
processing method, and storage medium containing the inventive program, weights of 
10 multiple picked-up images of a target object are computed, and averages of these 
images are calculated. A weighted average is then computed on the basis of the 
computed weights and of the averages of the multiple images, whereby an optimum 
image of the target object is created. 

Other objects, features and advantages of the invention will become more 
15 apparent upon a reading of the following description and appended drawings. 

BRIEF DESCRIPTION OF THE FIGURES 
Fig. 1 is a schematic view of one embodiment of this invention; 
Fig. 2 is a schematic view showing a typical structure of an optical system in 
an image pickup device included in Fig. 1 ; 
20 Fig. 3 is a block diagram depicting hardware typical of an image processing 

apparatus included in Fig. 1; 

Fig. 4 is a schematic view indicating regions in which a double integral of (u, 
v) with expression (4) yields 1 ; 
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Fig. 5 is a schematic view showing a region to which values computed by use 
of expression (5) apply; 

Fig. 6 is a schematic view depicting a region to which values computed by use 
of expression (4) apply; 
5 Fig. 7 is a schematic view illustrating another region to which values 

computed by use of expression (4) apply; 

Fig. 8 is a schematic view indicating regions to which values computed by use 
of expression (9) apply; 

Fig. 9 is a schematic view showing regions to which values computed by use 
10 of expression (11) apply; and 

Fig. 10 is a flowchart of steps constituting an image touch-up process. 

DETAILED DESCRIPTION OF THE INVENTION 
Fig. 1 is a schematic view of one embodiment of this invention. For purposes 
of description, an image pickup device 2 and an image processing apparatus 3 are 
1 5 shown separately in Fig. 1 . In practice, they may be integrated in a single enclosure. 
A target object 1 is an object of which images are picked up. The image 
pickup device 2 includes CCDs (charge-coupled devices) 12-1 through 12-3 which 
have different distances from a lens 1 1 (see Fig. 2) and which pick up images of the 
target object 1. The image processing apparatus 3 creates a single focused image 
20 based on three images of the target object 1 picked up by the image pickup device 2. 

Fig. 2 is a schematic view showing a typical structure of an optical system in 
the image pickup device 2. Light coming from the target object 1 is directed by a lens 
11 onto image pickup planes 12-la, 12-2a and 12-3a of the CCDs 12-1, 12-2 and 12-3 
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respectively via half-mirrors, not shown. It is assumed that the lens 1 1 has a focal 
length Z, a diameter D, and a distance A to the target object 1 . 

The CCDs 12-1 through 12-3 are arranged so that distances (flange-back) 
differ between a mount reference plane of the lens 1 1 on the one hand and the image 
5 pickup planes 12-la through 12-3a on the other hand. Thus positioned, the CCDs 
yield a plurality of picked-up images. It is assumed that reference characters B], B 2 
and B3 denote respectively the distances between the reference surface of the lens 1 1 
on the one hand, and the image pickup planes 12-la, 12-2a and 12-3a of the CCDs 12- 
1, 12-2 and 12-3 on the other hand. It is also assumed that reference character B 
1 0 stands for the distance between the reference surface of the lens 1 1 and a focal point Q 
of the target object 1 and that the relationship Bi < B 2 < B3 < B exists. 

The distances Bi through B 3 are fixed and remain constant regardless of the 
distance A. The difference between the distances Bi and B 2 as well as the difference 
between the distances B 2 and B 3 is infinitesimal relative to the distance B. The 
1 5 distance B between the lens 1 1 and the focal point Q, it should be noted, varies 
depending on the distance A between the target object 1 and the lens 1 1 . 

On the image pickup plane 12-la of the CCD 12-1, an image considerably out 
of focus is formed. This image has a circle of confusion with a radius Ri . The image 
is equivalent to a sharply-focused image (an image picked up on the plane at the 
20 distance B from the lens 1 1) being subjected to a box filter with the radius Ri. 

The box filter with the radius R signifies a filter that provides l/(7iR 2 ) within 
the radius R from the origin of the circle and gives 0 in other regions. 
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On the image pickup plane 12-2a of the CCD 12-2 is formed an image that is 
less out of focus than the image picked up on the plane 12-la of the CCD 12-1, but 
out of focus nonetheless. This image has a circle of confusion with a radius R 2 . The 
image is equivalent to a sharply-focused image (an image picked up on the plane at 
5 the distance B from the lens 11) being subjected to a box filter with the radius R 2 . 

On the image pickup plane 12-3 a of the CCD 12-3 is formed an image that is 
less out of focus than the image picked up on the plane 12-2a of the CCD 12-2, but 
still out of focus. This image has a circle of confusion with a radius R 3 . The image is 
equivalent to a sharply- focused image (an image picked up on the plane at the distance 
10 B from the lens 11) being subjected to a box filter with the radius R3. 

As described, the image pickup device 2 picks up three images with circles of 
confusion having the radii Ri, R2 and R3, and outputs these images to the image 
processing apparatus 3. 

Since the difference between the distances Bi and B 2 is arranged to be 
1 5 infinitesimal as with the difference between the distances B 2 and B 3 , the radii Ri and 
R 2 are regarded as approximately equal, and so are the radii R 2 and R 3 . As shown in 
Fig. 2, there exists the relationship that Ri > R 2 > R3. 

Fig. 3 is a block diagram depicting a typical electrical hardware structure of 
the image processing apparatus 3. Referring to Fig. 3, a processing circuit 21 reads 
20 processing programs from a program memory 22 over a bus 25 and executes the 
programs for processing, as will be described later. The program memory 22 
accommodates the programs to be carried out by the processing circuit 21 as well as 
data that are needed by the circuit 21 during its operation. A data memory 23 stores 

6 



image data prior to processing and processed data (i.e., focused-image data) derived 
from processing by the processing circuit 21 . A frame memory 24 temporarily holds 
the processed data on their way from the data memory 23 to an image display unit 29. 
The processing circuit 21, program memory 22, data memory 23, and frame memory 

5 24 are interconnected via the bus 25, and also connected to an I/O interface 26. 

The I/O interface 26 is further connected to an I/O terminal 27 and an input 
unit 28 in addition to the image display unit 29. The I/O terminal 27 admits image 
data constituting a plurality of images picked up by the image pickup device 2, and 
allows processed data to be output to an external device. The input unit 28 is 

1 0 illustratively made up of a keyboard and a mouse, and the image display unit 29 is 
composed of an LCD (liquid crystal display), a CRT (cathode ray tube) or like 
equipment. The I/O interface 26 is connected as needed with a drive 30 that allows 
programs to be installed. The drive 30 may accommodate a magnetic disc 41, an 
optical disc 42, a magneto-optical disc 43, or a semiconductor memory 44. 

1 5 Prior to a description of the workings of the preferred embodiment, the images 

(represented herein as G(x, y)) supplied from the image pickup device 2 to the image 
processing apparatus 3 will be explained in detail below. 

First, consider an averaging computation to be performed on images G(x, y) by 
use of the following expression (1): 

20 H{x,y,T) = —V G(x + Tcos(e),y + Tsm(8))d0 ...(1) 

where, H(x, y, T) denotes an average of the images G(x, y) in all locations at a 
distance T from a location (x, y), and 9 represents an angle variable. In expression (1) 
above, values of the images G(x, y) shifted by (Tcos(O), Tsin(6)) each from the 

7 



location (x, y) are integrated from 0 to 2% with respect to 0. The result from the 
integration is divided by 2n to yield an average of the images G(x, y). The average of 
the images G(x, y) will thus be denoted as the image H(x, y, T) hereunder. 

Suppose now that images G(x, y) are out of focus by a radius R each with 
5 respect to a sharply-focused image F(x, y), i.e., that the images G(x, y) have a circle of 
confusion with the radius R each. Each image having the circle of confusion with the 
radius R will be referred to as the image G(x, y, R) in the description that follows. 

As mentioned above, the image G(x, y, R) having the circle of confusion with 
the radius R is equivalent to a sharply-focused image F(x, y) being subjected to a box 
10 filter with the radius R. In that case, the image G(x, y, R) is given by expression (2): 



where, S(R) denotes the inside of a circle with the radius R. That is, in expression (2) 
above, the value of a focused image F(x, y) being out of focus by (u, v) is doubly 
integrated inside the circle (S(R)) having the radius R. The result from the double 
1 5 integration is divided by tcR 2 to yield the value of the image G(x, y, R). 

The average of the image G(x, y, R) having the circle of confusion with the 
radius R is computed by substituting expression (2) for expression (1) above. The 
average is thus given by expression (3): 



where, H(x, y, T, R) denotes the average of the images G(x, y, R) each having the 
circle of confusion with the radius R in all locations at the distance T from the location 
(x, y), and 0 represents an angle variable. The average of the images G(x, y, R) each 



G{x,y,R) = -^\[ m F(x + u,y + v)dudv 



...(2) 




dudvF{x + T cos(#) + u,y + T sin(0) + v) 



20 



...(3) 
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having the circle of confusion with the radius R will thus be denoted as the image H(x, 
y, T, R) hereunder. 

As described, performing the computation of expression (1) on the images 
G(x, y, R) having the circle of confusion with the radius R each provides a new image 
5 H(x, y, T, R). This new image coincides with the image H(x, y, T, R) acquired by 
carrying out the computation of expression (3) on the sharply- focused image F(x, y). 

Now consider the degree of contributions made by values of points in the 
sharply- focused image F(x, y) to the values of the image H(x, y, T, R) computed by 
use of expression (3) above. In other words, consider calculating data H c (x, y, x+p, 
10 y+q, T, R) indicating how much the values of the image F(x, y) in a location (x+p, 
y+q) contribute to the values of the image H(x, y, T, R) in the location (x, y). 

The data H c (x, y, x+p, y+q, T, R) is obtained by computations with expression 
(3) on the assumption that the result of the computations on the sharply-focused image 
F(x, y) is 1 in the location (x+p, y+q) and 0 elsewhere. That is, the data H c (x, y, x+p, 
15 y+q, T, R) is given by expression (4) based on a delta function: 



where, the double integral of (u, v) yields 1 if the point with an X coordinate of 
x+Tcos(9)-(x+p) and a Y coordinate of y+Tsin(8)-(y+q) exists within the radius R and 
20 provides 0 if the point is not found therein. In other words, the result of the double 
integral is 1 if the point (x+p, y+q) exists within the radius R around the point 
(x+Tcos(9), y+Tsin(9)) and is 0 if the point does not exist within the specified radius 
R. 



H c (x,y,x + p,y + q,T,R) 




(y + Tsm(0) + v)-(y + q) 



-(4) 
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Fig. 4 schematically shows regions in which the double integral of (u, v) with 
expression (4) yields 1. In Fig. 4, the horizontal axis is regarded as the X-axis and the 
vertical axis as the Y-axis (the same applies as well to other figures subsequent to Fig. 
4). Referring to Fig. 4, regions 5 1 and 52 are each a circle with the radius R. The 
5 center of each circle is at a distance T from a point (x, y). A point (x+p, y+q) exists 
on the circumference of each circle. A line segment from the point (x, y) to the center 
of the region 51 has an angle of q>i with respect to the X-axis; a line segment from the 
point (x, y) to the center of the region 52 has an angle of cp 2 relative to the X-axis. The 
distance T is assumed to be greater than the radius R (T > R). 
10 Suppose that there is an angle 9 between the angles q>i and <p 2 defined above. 

In that case, the point (x+p, y+q) exists within the radius R around the point 
(x+Tcos(9), y+Tsin(8)). This means that expression (4) above may be transformed 
into expression (5) below: 

H c ( X ,y, X + P ,y + q ,T,R) = ^ ¥ (; XdO = - A) = ^ 

15 -(5) 
where, Acp represents q> 2 - <pi (shown in Fig. 4). Fig. 5 schematically shows a region to 

which the data H c (x, y, x+p, y+q, T, R) computed by use of expression (5) applies. 

In Fig. 5, a region 61 is a circle with the radius R. The center (T, 0) of the 

circle is at a distance T from the origin O. Points Pi and P 2 exist on the 

20 circumference, each point at a distance L from the origin O. The distance L represents 

the distance between the points (x, y) and (x+p, y+q) shown in Fig. 4 and is expressed 

as L = V(p 2 +q 2 ). Reference character Acp denotes the angle between the points Pi and 

P 2 on a circular arc with respect to the origin O. The distance L is subject to the 

relation of (T - R) < L < (T + R). 



If the distance T is greater than the radius R and if the distance L between the 
points (x, y) and (x+p, y+q) is subject to the relation of (T - R) < L < (T + R) as 
described, then it is possible to compute the data H c (x, y, x+p, y+q, T, R) in 
expression (4) above. Where the distance T is less than the radius R, it is also possible 
5 to compute likewise the data H c (x, y, x+p, y+q, T, R) in expression (4). 

Fig. 6 schematically depicts a region to which the data Hc(x, y, x+p, y+q, T, 
R) computed with expression (4) applies when the distance T is less than the radius R 
and when the distance L between the points (x, y) and (x+p, y+q) is subject to the 
relation of (R - T) < L < (T + R). 

10 In Fig. 6, a region 71 is a circle with the radius R. The center (T, 0) of the 

circle is at a distance T from the origin O. Points P 1 and P2 exist on the 
circumference, each point at a distance L from the origin O. The distance L represents 
the distance between the points (x, y) and (x+p, y+q) shown in Fig. 4 and is expressed 
as L = V(p 2 +q 2 ). Reference character Acp denotes the angle between the points Pi and 

15 P2 on a circular arc with respect to the origin O. 

Fig. 7 schematically illustrates another region to which the data H c (x, y, x+p, 
y+q, T, R) computed with expression (4) applies when the distance T is less than the 
radius R and when the distance L between the points (x, y) and (x+p, y+q) is subject 
to the relation of L < (R - T). 

20 In Fig. 7, a region 81 is a circle with the radius R. The center (T, 0) of the 

circle is at a distance T from the origin O. In this case, the angle Acp is 360 degrees 
because a point (p, q) exists within the radius R around a point (Tcos(0), Tsin(O)) at all 
angles 8 (i.e., 0 through 360 degrees). 



As can be seen in Figs. 5 through 7, the data H c (x, y, x+p, y+q, T, R) in 
expression (4) above is related not directly to values x, y, p and q but to the distance L 
between the points (x, y) and (x+p, y+q). For purpose of simplification, the data Hc(x, 
y, x+p, y+q, T, R) will be referred to as the data H C (L, T, R) in the description that 
follows. 

Consider the fact that the point (p, q) does not exist within the radius R around 
the point (Tcos(6), Tsin(6)) at all angles 6 (0 through 360 degrees) if T > R and L < (T 
- R) or if T > R and (T + R) < L. Given that fact, expression (4) above may be 
replaced by expression (6) below: 
360 



10 H c (L,T,R) = -^- 2 



Zd2 degrees (0<L<(R-T) 

cos~ 1 [Jm-{m-r)ilit} {\T-R\<L<T + R 



2n 2 R 
2 



7T 2 R 

0 degree (elsewhere 

-(6) 

The value M in expression (6) above is provided by expression (7): 

M = (L + R + T)/2 ...(7) 
As described, the new image H(x, y, T, R) can be obtained by performing 
15 computations of expression (1) on the images G(x, y, R) having the circle of confusion 
with the radius R each. This new image coincides with the image H(x, y, T, R) 
acquired by carrying out the computation of expression (3) on the sharply-focused 
image F(x, y). 

The value of the image H(x, y, T, R) in a given location (x, y) is acquired by 
20 weighting and adding up (i.e., integrating) the values of the images F(x, y) in locations 
(x+p, y+q) as shown in expression (3) above. The weight (i.e., contribution ratio) 



used here is a function of L =V(p 2 +q 2 ). The value of the function is equivalent to the 
data Hc(L, T, R) in expression (6) above. 

Now consider three out-of-focus images with different degrees of defocusing 
with regard to a sharply- focused image F(x, y). The degrees of defocusing of the three 
5 images are represented by Ri, R2 and R3 (Ri > R2 > R3; expressed by radii of their 
circles of confusion). The three images are thus defined as Gi(x, y, R^, G 2 (x, y, R 2 ) 
and G 3 (x, y, R 3 ). Computations with expression (3) are carried out on these images 
Gi(x, y, Ri), G 2 (x, y, R 2 ) and G3(x, y, R 3 ). It is assumed that T = R 2 for the image 
Gi(x, y, Ri), that T =Ri for the image G 2 (x, y, R 2 ) and that T = Rj + R 2 - R 3 for the 
10 image G 3 (x, y, R 3 ). 

Performing the computations with expression (3) provides an image H(x, y, 
R 2 , Ri) as an average of the image Gi(x, y, Ri), an image H(x, y, Ri, R 2 ) as an average 
of the image G 2 (x, y, R 2 ) and an image H(x, y, Ri + R 2 - R 3 , R 3 ) as an average of the 
image G 3 (x, y, R 3 ). These three images are subjected to computations with expression 
15 (8) below for a weighted average using weights Wi, W 2 and W 3 : 
H 3 (x,y,R 1 ,R2^3,Wi,W2,W 3 ) 

=Wi-H(x,y,R 2 ,Ri)+W 2 -H(x,y,Ri,R 2 )+W 3 -H(x,y,Ri+R 2 -R 3 ,R 3 ) 

...(8) 

where, H 3 (x, y, R l5 R 2 , R 3 , Wi, W 2 , W 3 ) denotes a weighted average of the three 
20 images H(x, y, R 2 , Ri), H(x, y, R u R 2 ) and H(x, y, Ri + R 2 - R 3 , R 3 ). This weighted 
average will be abbreviated as the image H 3 hereunder. As described, the value of the 
image H 3 in a given location (x, y) is obtained by weighting and adding up (i.e., 
integrating) the values of the images F(x, y) in locations (x+p, y+q). The weight used 
here (i.e., contribution rate) is given by expression (9) below: 



H 3C (L,Ri ,R 2 ,R 3 , Wi , W 2 , W 3 ) 

=WrHc(L,R2,Ri)+W2-Hc(L,Ri,R 2 )+W3-Hc(L,Ri+R2-R3,R3) 

...(9) 

where, H 3C (L, Ri, R 2 , R 3 , Wi, W 2 , W 3 ) denotes the weight of the image H 3 . This 
5 weight will be referred to as the weight H 3C in the description that follows. Reference 
character L represents the distance between the points (x, y) and (x+p, y+q) shown in 
Fig. 4 and is expressed as L = V(p 2 +q 2 ). 

Fig. 8 schematically indicates regions to which the weight H 3C computed by 
use of expression (9) applies. In Fig. 8, regions 91 through 93 are circles with radii 
10 Ri, R 2 and R 3 around centers (R 2 , 0), (R b 0) and (R t + R 2 - R 3 , 0) respectively. 
Consider now a distance L] in Fig. 8 where the distance L is less than (Ri - R 2 ). 

In the case above, the first term WrHc(L, R 2 , Ri) on the right-hand side in 
expression (9) above corresponds to the region of 0 < L < (R - T) in expression (6), as 
explained with reference to Fig. 7. Thus the term is expressed as 
15 WrH c (L, R 2 , Ri) = \yV(360/2(jrRi) 2 ) 

The second term W 2 -H C (L, R u R 2 ) and the third term W 3 -H C (L, Ri + R 2 - R 3 , R 3 ) on 
the right-hand side in expression (9) are both zero. 

Consider now a distance L 2 in Fig. 8 where the distance L is subject to the 
relation of (Ri - R 2 ) < L < (Ri + R 2 ) - (2-R 3 ). It is assumed that the angle of a circular 
20 arc formed by points Ni, N 2 , N 3 and N4 with respect to the origin O is represented by 
ZNiN 2 N 3 N4 and that the angle of a circular arc formed by the points N 2 and N 3 
relative to the origin O is denoted by ZN2N3. 

In the case above, the first term WrH c (L, R 2 , Ri) on the right-hand side in 
expression (9) is expressed as 



Wi-H C (L, R 2 , Ri) = Wr(ZNiN2N 3 N4/2(icRi) 2 ) 
The second term W 2 -H C (L, Ri, R 2 ) on the right-hand side in expression (9) is 
expressed as 

W 2 -H C (L, R 1; R 2 ) = W 2 -(ZN 2 N 3 /2(7tR 2 ) 2 ) 
5 The third term W 3 -H C (L, Ri + R 2 - R3, R3) on the right-hand side in expression (9) is 
zero. 

Consider a distance L3 in Fig. 8 where the distance L is subject to the relation 
of (Ri + R 2 - R 3 ) < L < (Ri + R 2 ). It is assumed that the angle of a circular arc formed 
by points N 5 , N 6 , N 7 , N 8 , N 9 and Ni 0 with respect to the origin O is represented by 
1 0 ZN5N6N7N8N9N10, that the angle of a circular arc formed by the points N 6 , N 7 , N 8 and 
N 9 relative to the origin O is denoted by ZNeNjNgNg and that the angle of a circular 
arc formed by the points N 7 and N 8 with regard to the origin O is represented by 
ZN 7 N 8 . 

In the case above, the first term WrH c (L, R 2 , Ri) on the right-hand side in 
15 expression (9) is expressed as 

WrHc(L, R 2 , Ri) = WrCZNsNe^NgNgNio^CTiRO 2 ) 
The second term W 2 -H C (L, R u R 2 ) on the right-hand side in expression (9) is 
expressed as 

W 2 -H C (L, R u R 2 ) = W 2 -(ZN 6 N7N 8 N9/2(7rR2) 2 ) 
20 The third term W3-Hc(L, R\ + R 2 - R3, R3) on the right-hand side in expression (9) is 
expressed as 

W 3 -H C (L, Ri + R 2 - R 3 , R 3 ) = W 3 -(Z 7 N 8 /2(jiR3) 2 ) 
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If the distance L is subject to the relation of (Ri + R 2 ) < L, then all terms 
on the right-hand side in expression (9) are zero. 

As shown in Fig. 8, if illustratively Wi = (tiRi) 2 , W 2 = -2-(7iR 2 ) 2 and W 3 = 
(7iR 3 ) 2 and if L = L L , then the value of the weight H 3C is expressed as 
5 H 3C = 360/2= 180 

If L = L 2 , the value of the weight H 3C is expressed as 

H 3C = (ZNiN 2 N 3 N4/2) - 2-(ZN 2 N 3 /2) 
This value is approximately zero as can be seen from Fig. 8. If L = L 3 , the value of 
the weight H 3 c is expressed as 
10 H 3C = (ZN 5 N 6 N 7 N8N 9 Nio/2) - 2-(ZN 6 N 7 N8N 9 /2) + (ZN 7 N 8 /2) 

This value is virtually zero as can be seen from Fig. 8. 

With the weights Wi, W 2 and W 3 established as described, the weight H 3C of 
the three images becomes extremely close to zero if the distance L is equal to or 
greater than (Rj - R 2 ), unlike in the case where the distance L is less than (Ri - R 2 ). It 
15 can thus be understood that only the value of the sharply- focused image F(x, y) near 
the location (x, y) contributes to the value of the image H 3 in the location (x, y). 

Consider now two out-of- focus images with different degrees of defocusing 
with regard to a sharply- focused image F(x, y). The degrees of defocusing of the two 
images are represented by Ri and R 2 (Rj > R 2 ; expressed by radii of their circles of 
20 confusion). The two images are thus defined as Gi(x, y, Ri) and G 2 (x, y, R 2 ). 

Computations with expression (3) are carried out on these images Gi(x, y, Ri) and 
G 2 (x, y, R 2 ). It is assumed that T = R 2 for the image Gi(x, y, Ri) and T = Ri for the 
image G 2 (x, y, R 2 ). 
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Performing the computations with expression (3) provides an image H(x, y, 
R2, Ri) as an average of the image Gi(x, y, Ri) and an image H(x, y, R l5 R 2 ) as an 
average of the image G 2 (x, y, R 2 ). These two images are subjected to computations 
with expression (10) below for a weighted average using weights Wi and W 2 : 
5 H 2 (x,y,Ri,R 2 ,Wi,W2)=WrH(x,y,R2,Ri)+W 2 -H(x,y,R 1 ,R 2 ) 

...(10) 

where, H 2 (x, y, Ri, R 2 , Wi, W 2 ) denotes a weighted average of the two images H(x, y, 
R 2 , Ri) and H(x, y, R ls R 2 ). This weighted average will be abbreviated as the image 
H 2 hereunder. As described, the value of the image H 2 in a given location (x, y) is 
10 obtained by weighting and adding up (i.e., integrating) the values of the images F(x, y) 
in locations (x+p, y+q). The weight used here (i.e., contribution rate) is given by 
expression (11) below: 

H 2C (L,R 1 ,R 2 ,W 1 ,W 2 )=Wi-Hc(L,R 2 ,R 1 )+W 2 -Hc(L,R 1 ,R 2 ) 

...(11) 

1 5 where, H 2C (L, Ri, R 2 , Wi, W 2 ) denotes the weight of the image H 2 . This weight will 
be referred to as the weight H 2C in the description that follows. Reference character L 
represents the distance between the points (x, y) and (x+p, y+q) shown in Fig. 4 and is 
expressed as L = V(p 2 +q 2 ). 

Fig. 9 schematically depicts regions to which the weight H 2C computed by use 

20 of expression (1 1) applies. In Fig. 9, regions 101 through 102 are circles with radii Ri 
and R 2 around centers (R 2 , 0) and (Ri, 0) respectively. Consider now a distance L 4 in 
Fig. 9 where the distance L is less than (Ri - R 2 ). 
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In the case above, the first term Wi-H C (L, R 2 , Ri) on the right-hand side in 
expression (11) above corresponds to the region of 0 < L < (R - T) in expression (6), 
as explained with reference to Fig. 7. Thus the term is expressed as 
Wi-H C (L, R 2 , RO = Wi-(360/2(zR!) 2 ) 
5 The second term W 2 -H C (L, R u R 2 ) on the right-hand side in expression (1 1) is zero. 

Consider now a distance L 5 in Fig. 9 where the distance L is subject to the 
relation of (R t - R 2 ) < L < (R x + R 2 ). It is assumed that the angle of a circular arc 
formed by points Ni i, Nj 2 , N i3 and N J4 with respect to the origin O is represented by 
ZN11N12N13N14 and that the angle of a circular arc formed by the points Ni 2 and N13 
1 0 relative to the origin O is denoted by ZNi 2 N 13 . 

In the case above, the first term WrH c (L, R 2 , Ri) on the right-hand side in 
expression (11) is expressed as 

WrH c (L, R 2 , Ri) = WKZNuNnNnNi^TtRO 2 ) 
The second term W 2 -H C (L, R l9 R 2 ) on the right-hand side in expression (1 1) is 
15 expressed as 

W 2 -H C (L, R b R 2 ) = W 2 -(ZN 12 N 13 /2(7rR 2 ) 2 ) 
If the distance L is subject to the relation of (Ri + R 2 ) < L, then all terms on the 
right-hand side in expression (1 1) are zero. 

As shown in Fig. 9, if illustratively Wi - (tiR^ 2 and W 2 = -C-(ttR 2 ) 2 and if L = 
20 L 4 , then the value of the weight H 2C is expressed as 
H 2C = 360/2= 180 

where, C stands for a value of about 1.2. If L = L 5 , then the value of the weight H 2C is 
expressed as 
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H 2C = (ZNiiNi 2 N 13 Ni4/2) - C-(ZNi 2 Ni 3 /2) 
This value is virtually zero as can be seen from Fig. 9. 

With the weights Wi and W 2 established as described, the weight H 2C of the 
two images becomes extremely close to zero if the distance L is equal to or greater 
5 than (Ri - R 2 ), unlike in the case where the distance L is less than (Ri - R 2 ). It can 
thus be understood that only the value of the sharply-focused image F(x, y) near the 
location (x, y) contributes to the value of the image H 2 in the location (x, y). 

An image touch-up process performed by the image processing apparatus 3 
will now be described with reference to the flowchart of Fig. 10. The process involves 
10 creating a sharply- focused image by touching up three out-of-focus images with 
different degrees of defocusing. 

In step SI of Fig. 10, the processing circuit 21 reads from the image pickup 
device 2 image data representing a plurality of images (3 images in this example) to be 
touched up (i.e., to be brought into focus). The image data are held temporarily in the 
15 data memory 23 following input through the I/O terminal 27, I/O interface 26 and bus 
25. Three images Gi(x, y), G 2 (x, y) and G 3 (x, y) are now placed in the data memory 
23. 

In step S2, the input unit 28 is used to input degrees of defocusing (i.e., radii of 
circles of confusion) with respect to the data representative of the three images read in 
20 step SI. Specifically, a user operates the input unit 28 to enter different degrees of 
defocusing regarding the three images. This establishes three images Gi(x, y, Ri), 
G 2 (x, y, R 2 ) and G 3 (x, y, R 3 ). 

In step S3, the processing circuit 21 reads the data of the three images from the 

data memory 23 to compute weights Wi, W 2 and W 3 . The weights Wi, W 2 and W 3 
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are computed by the processing circuit 21 in such a manner that the weight H 3C shown 
in expression (9) is minimized on condition that L > (Ri - R 2 ). The processing circuit 
21 stores the computed weights Wi, W 2 and W 3 into the data memory 23. 

In step S4, the processing circuit 21 computes an average of each of the three 
5 images whose data are held in the data memory 23 . Specifically, the processing 

circuit 21 computes averages of the images Gi(x, y, Ri), G 2 (x, y, R 2 ) and G 3 (x, y, R 3 ) 
using expressions (12), (13) and (14) respectively: 

K 0> y) = — C G x O + R 2 cos(0), y + R 2 sin(0))d0 ...(12) 

h 2 (x,y) = — (* G 2 (x + R l cos(6),y + R l sm(6))d0 ...(13) 
2ji * 

10 h 3 (x, y) = — 1* G 3 (x + (R 1 +R 2 -R 3 )- cos(0), y + (R x + R 2 - R 3 ) ■ sin(0))d9 

2?T 

...(14) 

With expression (12), values of the image Gi(x, y, Ri) shifted by (R 2 cos(9), 
R2sin(6)) from the location (x, y) are integrated from 0 to 2n with respect to 0. The 
result from the integration is divided by 2n to give the average of the image Gi(x, y, 

15 Ri). With expression (13), values of the image G2(x, y, R 2 ) shifted by (Ricos(6), 
Risin(9)) from the location (x, y) are integrated from 0 to 2n with respect to 0. The 
result from the integration is divided by 2n to give the average of the image G 2 (x, y, 
R 2 ). With expression (14), values of the image G3(x, y, R 3 ) shifted by ((Ri + R 2 - 
R 3 )cos(0), (Ri + R 2 - R 3 )sin(0)) from the location (x, y) are integrated from 0 to 2n 

20 with respect to 0. The result from the integration is divided by 2n to give the average 
of the image G 3 (x, y, R 3 ). 

In step S5, the processing circuit 21 performs computations with expression 
(15) below on the averages of the three-image data computed in step S4, i.e., on 



images h^x, y), h 2 (x, y) and h 3 (x, y) to acquire a weighted average using the weights 
Wi, W 2 and W 3 calculated in step S3: 

H 3 (x,y)=W 1 -h 1 (x,y)+W 2 -h 2 (x,y)+W3-h 3 (x,y) ...(15) 

With expression (15), the images hi(x, y), h 2 (x, y) and h 3 (x, y) are weighted by 
Wi, W 2 and W 3 respectively before being added up to provide an image H 3 . 

In step S5, the processing circuit 21 stores the computed image H 3 into the 
data memory 23. 

In step S6, the processing circuit 21 reads the image H 3 that was computed and 
placed into the data memory 23 in step S5, and places the retrieved image temporarily 
into the frame memory 24. From the frame memory 24, the processing circuit 21 
outputs the image to the image display unit 29 through the bus 25 and I/O interface 
26. This terminates the image touch-up process. 

As described, a sharply- focused image is obtained by subjecting three out-of- 
focus images with different degrees of defocusing (i.e., radii of different circles of 
confusion) to weighted averaging. 

In the process discussed above, the radii Ri, R 2 and R 3 were assumed to be 
known so that the user could make their entries from the input unit 28. Generally, 
however, these values are unknown. In such cases, the radii R u R 2 and R 3 may be 
suitably varied in values when subjected repeatedly to the process above until an 
optimum image H 3 is obtained. 

To obtain the optimum image H 3 at this point requires performing one of two 
proceedings: either the user checks to see if an image on the image display unit 29 is 
optimum (i.e., visually checks the image if it is focused); or the displayed image is 
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automatically checked for any high-frequency components. The latter check takes 
advantage of the fact that out-of-focus images contain no high-frequency components. 

Meanwhile, as shown in Fig. 2, the radii R u R 2 and R 3 are subject to the 
relation defined by expression (16): 
5 Ri:R 2 :R 3 :(D/2)=(B-Bi):(B-B2):(B-B3):B ...(16) 

where, B stands for an indefinite value. In that case, only those radii R h R 2 and R 3 
which satisfy the relation defined by expression (16) may be allowed to undergo the 
image touch-up process. 

If the distance A between the target object 1 and the lens 1 1 is known, the 
10 distance B may be computed by use of expression (17): 

(1/A) + (1/B) = (1/Z) ...(17) 
The radii Rj, R 2 and R3 are thus computed (i.e., established) by expressions 
(16) and (17) above. 

The ratio of the radii Ri, R 2 and R 3 may be computed by analyzing the images 
15 Gi(x, y), G 2 (x, y) and G 3 (x, y). As explained earlier, the images Gi(x, y), G 2 (x, y) and 
G 3 (x, y) are images that have undergone filtering by box filters with radii Ri, R 2 and 
R 3 . Illustratively, when an image G is subject to a box filter with a radius R, the 
component of a frequency co is defined by expression (18): 

2-R-sinc(caR/7c) = 2-R-sin(coR)/coR ...(18) 
20 Hence the following computations: the images Gi(x, y), G 2 (x, y) and G 3 (x, y) 

are subjected to frequency division to calculate the component co of each of their 
frequencies. Then the ratio of the radii Ri, R 2 and R 3 need only be determined in such 
a manner that the ratio of the components at the respective frequencies satisfies 
expression (19) below: 



Ri • sinc(caR 1 /7r):R 2 -sinc(cDR2/7i) :R 3 -sinc(o)R3/7c) 

...(19) 

The ratio of the radii R b R 2 and R 3 is thus computed (i.e., established) by 
use of expression (19). 
5 As described, there is no need to subject all radii Ri, R 2 and R 3 to the 

image touch-up process. That is, the process need only be carried out on the radii Ri, 
R 2 and R 3 computed with expressions (16) and (17) above, or on the radii Ri, R 2 and 
R 3 which satisfy expression (19). 

Another image touch-up process performed by the image processing 
10 apparatus 3 is described below by again referring to the flowchart of Fig. 10. This 
time, the process involves creating a sharply- focused image by touching up two out- 
of-focus images with different degrees of defocusing. 

In step SI of Fig. 10, the processing circuit 21 reads from the image 
pickup device 2 image data representing a plurality of images (two images in this 
15 example) to be touched up (i.e., to be brought into focus). The image data are held 
temporarily in the data memory 23 following input through the I/O terminal 27, I/O 
interface 26 and bus 25. Two images Gi(x, y) and G 2 (x, y) are now placed in the data 
memory 23. 

In step S2, the input unit 28 is used to input degrees of defocusing (i.e., 
20 radii of circles of confusion) with respect to the data representative of the two images 
read in step SI . Specifically, the user operates the input unit 28 to enter different 
degrees of defocusing regarding the two images. This establishes two images Gi(x, y, 
Ri) and G 2 (x, y, R 2 ). 
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In step S3, the processing circuit 21 reads the data of the two images from 
the data memory 23 to compute weights Wi and W 2 . The weights Wi and W 2 are 
computed in such a manner that the weight H 2C shown in expression (1 1) is minimized 
on condition that L > (Ri - R 2 ). The processing circuit 21 stores the computed weights 
5 Wi and W 2 into the data memory 23. 

In step S4, the processing circuit 21 computes an average of each of the 
two images whose data are held in the data memory 23. Specifically, the processing 
circuit 21 computes averages of the images Gi(x, y, Ri) and G 2 (x, y, R 2 ) using 
yK expressions (12) and (13) respectively. 

HI 10 In step S5, the processing circuit 21 performs computations with 

i]l expression (20) below on the averages of the two-image data computed in step S4, i.e., 

I., s on images hi(x, y) and h 2 (x, y) to acquire a weighted average using the weights Wi 

il 

u, and W 2 calculated in step S3 : 

H 2 (x,y)=W 1 -h 1 (x,y)+W 2 -h 2 (x,y) ...(20) 
15 With expression (20) above, the images hi(x, y) and h 2 (x, y) are weighted by 

Wi and W 2 respectively before being added up to provide an image H 2 . 

In step S5, the processing circuit 21 stores the computed image H 2 into the 
data memory 23. 

In step S6, the processing circuit 21 reads the image H 2 that was computed and 
20 placed into the data memory 23 in step S5, and places the retrieved image temporarily 
in the frame memory 24. From the frame memory 24, the processing circuit 21 
outputs the image to the image display unit 29 through the bus 25 and I/O interface 
26. This terminates the image touch-up process. 



As described, a sharply-focused image is obtained by subjecting two out-of- 
focus images with different degrees of defocusing (i.e., radii of different circles of 
confusion) to weighted averaging. 

Alternatively, a sharply-focused image may be acquired by subjecting to 
5 weighted averaging not two or three but four or more out-of- focus images with 
different degrees of defocusing. 

In such cases where multiple out-of- focus images with different degrees of 
defocusing are used to create a sharply- focused image, the weighted average involved 
is generally calculated by expression (21) below: 

...(21) 

where, H K (x, y) denotes the weighted average of images Gj(x, y) having as many as k 
different circles of confusion (with radii Rj = 1 through k). In expression (22) below, 
the distance T y and weight Wy- are set to be approximately zero when the value L is 
15 sufficiently large: 

H KC {L,{T d }) = g "ff W, ■ H C (L,T J! ,R I ) ...(22) 

where, H K c(L, {Ty}) represents the weight of the image H K (x, y). The right-hand term 
in expression (22) above is defined by expression (6) given earlier. 

Illustratively, given images with three different circles of confusion, the right- 
20 hand term is defined so that k = 3, J(l) = 1, J(2) = 1, J(3) = 1, Tn = R 2 , T 2i = Ri, T 3J = 
Ri + R 2 - R 3 , W u = Wi = (ttRO 2 , W 21 = W 2 = -2-(ttR 2 ) 2 , and W 3 i = W 3 = (ttR 3 ) 2 . 
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Given images with two different circles of confusion, the right-hand term 
above is defined so that k = 2, J(l) = 1, J(2) = 1, Tn = R 2) T 2 i = Ri, Wn=Wi = 
(tcRi) 2 , and W 2 i = W 2 = -1.2-(tiR 2 ) 2 . 

In the manner described, a sharply-focused image is obtained on the basis of 
5 multiple out-of-focus images with different degrees of defocusing without movement 
of the lens. 

In a preferred setup, the red, green and blue components of the light entering 
the lens 1 1 may be arranged to undergo weighted averaging as different images with 
different degrees of defocusing. The arrangement, if implemented, allows for lenses 
10 whose chromatic aberration is appreciable and cannot be ignored. 

When a picture of a scene with different depths is taken, the picture may be 
divided into image segments whose varied depths are regarded as different degrees of 
defocusing. Subjecting these image segments to weighted averaging creates what is 
known as a pan-focused image (i.e., an image depicting the nearest object through the 
1 5 farthest being all in focus). 

The series of steps described above may be executed either by hardware or by 
software. For software-based processing to take place, programs constituting the 
software may be either incorporated beforehand in dedicated hardware or installed 
upon use from a suitable program storage medium into a general-purpose personal 
20 computer or like equipment capable of executing diverse functions. 

As shown in Fig. 3, the storage medium is offered to users apart from the 
computer not only as a package medium constituted by the magnetic disc (including 
floppy discs) 41, magneto-optical disc (including MD (Mini-disc), a registered 
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trademark) 42, optical disc (including CD-ROM (compact disc-read only memory) 
and DVD (digital versatile disc)) 43, or semiconductor memory 44; but also in the 
form of the program memory 22 or like built-in memory containing the programs and 
incorporated beforehand in the computer. 
5 In this specification, the steps which are stored on a program storage medium 

and which describe the programs to be executed represent not only processes that are 
carried out in the depicted sequence (i.e., on a time series basis) but also those that are 
conducted parallelly or individually. 

As described and through the use of the inventive image processing apparatus, 

10 image processing method, and storage medium containing the inventive program, 

weights of multiple picked-up images of a target object are computed, and averages of 
these images are calculated. A weighted average is then computed on the basis of the 
computed weights and of the averages of the multiple images, whereby an optimum 
image of the target object is created. The apparatus, method and program according to 

15 the invention thus permit creation of sharply- focused images at low cost. 

As many apparently different embodiments of this invention may be made 
without departing from the spirit and scope thereof, it is to be understood that the 
invention is not limited to the specific embodiments thereof except as defined in the 
appended claims. 
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